Adversarial Imagination Priors

Given an image, humans effortlessly run the image formation process backwards in their minds: they can tell albedo from shading, foreground from background, and imagine the occluded parts of the scene behind foreground objects. In this work, we propose a weakly supervised inversion machine trained to generate similar imaginations that when rendered using differentiable, graphics-like decoders, produce the original visual input. We constrain the imagination spaces by providing exemplar memory repositories in the form of foreground segmented objects, albedo, shading, background scenes and imposing adversarial losses on the imagination spaces. Our model learns to perform such inversion with weak supervision, without ever having seen paired annotated data, that is, without having seen the image paired with the corresponding ground-truth imaginations. We demonstrate our method by applying it to three Computer Vision tasks: image in-painting, intrinsic decomposition and object segmentation, each task having its own differentiable renderer. Data driven adversarial imagination priors effectively guide inversion, minimize the need for hand designed priors of smoothness or good continuation, or the need for paired annotated data. Consider Figure 1. We imagine a missing triangle occluding three small black circles rather than three carefully arranged pacman shapes – which is what the pixels depict. In (b), we do not perceive two parts of the sea separated by a standing person, rather a continuous sea landscape. In (c), we explain the input as a ”masked 8” rather than two semicircles. Consistent explanations of visual observations in terms of familiar concepts and memories we call “imaginations”. Imaginations invert the image formation process and propose 3D shape, camera pose, scene layering, spatial layout, albedo, shading, inpainted, un-occluded perceptions of the world, necessary for the understanding of the visual scene and interaction with it. Gestalt philosophers (Smith (1988)) proposed a set or principles to explain formation of such percepts, such as, closure, center surround pop-out, good continuity, smoothness etc, which many works attempt to hand design principles to incorporate those into computational frameworks of e.g., perceptual grouping (Yu (2003)). In this work, we present a learning-based inversion model that uses data-driven priors instead. We propose a computational model that addresses inverse problems in Computer Vision using adversarial imagination priors. Figure 2 illustrates our model. It is comprised of a generator neural network that given a visual input predicts visual imaginations, such as, in-painted image, un-occluded background scene, object segmentation, albedo and shading etc. Relevant memories, assumed to +